Consistent Risk Estimation in Moderately High-Dimensional Linear Regression

Risk estimation is at the core of many learning systems. The importance this problem has motivated researchers to propose different schemes, such as cross validation, generalized and Bootstrap. theoretical properties estimators have been extensively studied in low-dimensional settings, where number predictors p much smaller than observations n. However, a unifying methodology accompanied with rigorous theory lacking high-dimensional settings. This paper studies risk under moderately asymptotic setting n,p → ∞ n/p δ > 1 ( fixed number), proves consistency three that successful numerical studies, i.e., leave-one-out validation (LOOCV), approximate (ALO), message passing (AMP)-based techniques. A corner stone our analysis bound we obtain on discrepancy `residuals' obtained from AMP LOOCV. connection not only enables us more refined information estimates AMP, ALO, LOOCV, but also offers an upper convergence rate each estimator.